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Today,  programs  are  required  to  do  more  with  less. 
With  70  percent  of  a  system’s  life-cycle  cost  set  at  pre- 
Milestone  B,  the  most  significant  cost  savings  potential  is 
prior  to  Milestone  B.  Pre-Milestone  B  efforts  are  usually 
reduced  to  meet  tight  program  schedules.  This  article 
proposes  a  new  Systems  Engineering  Concept  Tool 
and  Method  (SECTM)  that  uses  genetic  algorithms  to 
quickly  identify  optimal  solutions.  Both  are  applied  to 
unmanned  undersea  vehicle  design  to  show  process 
feasibility.  The  method  increases  the  number  of  alter¬ 
natives  assessed,  considers  technology  maturity  risk, 
and  incorporates  systems  engineering  cost  into  the 
Analysis  of  Alternatives  process.  While  not  validated, 
the  SECTM  would  enhance  the  likelihood  of  success  for 
sufficiently  resourced  programs. 
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This  article  examines  the  importance  of  developing  a  robust  Analysis 
of  Alternatives  (AoA)  early  in  the  concept  phase  of  the  acquisition  pro¬ 
gram  (prior  to  Milestone  B)  and  the  effects  such  development  may  have 
on  program  success.  While  current  statutes  require  that  program  man¬ 
agers  complete  an  AoA  for  all  Acquisition  Category  (ACAT)  programs, 
the  quality  of  the  AoA  is  the  predominant  indicator  for  program  suc¬ 
cess  and  consists  of  more  than  just  completing  a  study  (Government 
Accountability  Office  [GAO],  2009a).  In  2008,  the  Department  of  Defense 
(DoD)  had  96  major  defense  acquisition  programs,  which  experienced  a 
cost  growth  of  $296  billion  and  an  average  schedule  delay  of  22  months 
(GAO,  2009a).  The  GAO  completed  a  study  in  2009  where  it  identified 
one  of  the  key  causes  for  this  cost  and  schedule  growth  as  the  mismatch 
between  the  requirements  of  the  systems  and  the  resources  to  provide 
them  (GAO,  2009a).  GAO  further  stated  that  programs  enter  the  acquisi¬ 
tion  process  with  requirements  that  are  not  fully  understood,  cost  and 
schedule  estimates  that  are  based  on  optimistic  assumptions,  and  a  lack 
of  sufficient  knowledge  about  technology,  design,  and  manufacturing. 


While  SECTM  cannot  be  validated  until 
implemented  by  acquisition  programs,  it  is 
expected  to  increase  the  likelihood  of  successful 
programs  that,  if  sufficiently  resourced,  can 
deliver  on  time  and  on  budget. 


The  DoD  has  a  history  of  rushing  programs  into  development  or 
production  that  are  not  ready  due  to  various  program  constraints.  The 
Joint  Strike  Fighter  was  intended  to  produce  an  affordable  aircraft,  but 
ended  up  being  the  most  expensive  aircraft  program  in  DoD  with  over 
$200  billion  for  3,000  aircraft.  GAO  attributed  a  major  factor  for  the 
cost  overrun  to  the  program’s  premature  entry  into  the  engineering, 
manufacturing,  and  development  phase  prior  to  the  maturation  of  criti¬ 
cal  technologies  (GAO,  2001).  The  Navy  has  entered  into  shipbuilding 
contracts  without  fully  maturing  component  technologies,  resulting 
in  a  193  percent  cost  growth  on  Littoral  Combat  Ship  1  and  a  52-month 
delay  on  Landing  Platform  Dock  17  (GAO,  2009b).  This  rush  is  not  just 
on  large  ACAT  I  programs,  but  also  on  smaller  ACAT  programs  (Pincus, 
2012).  The  Navy  Organic  Airborne  and  Surface  Influence  Sweep  (OASIS) 
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program  just  experienced  a  cost  increase  from  $55  million  to  $135  mil¬ 
lion,  with  an  8-year  delay  in  fielding.  This  system  still  has  not  met  the 
requirement  to  continue  operating  after  being  hit  by  a  shock  wave  from 
a  mine  or  ordnance  explosion  (Pincus,  2012).  The  latest  results  from  the 
last  Department  of  Defense  Inspector  General  (DoDIG)  study  indicated 
OASIS  met  only  65  percent  of  its  shock  requirement  and  would  not  work 
(DoDIG,  2012). 

This  article  defines  a  new  Systems  Engineering  Concept  Tool  and 
a  five-step  system  engineering  Method  (SECTM)  that  we  developed  to 
increase  the  robustness  of  AoAs.  We  based  the  SECTM  design  on  the 
finding  from  the  GAO  (2009a)  study  that  examined  32  DoD  programs, 
and  the  impacts  that  the  quality  of  the  AoA  can  have  on  program  success. 
We  applied  SECTM  to  a  UUV  concept  design  to  show  the  feasibility  of 
implementing  the  process.  SECTM  includes  a  Systems  Engineering 
Concept  Tool  based  off  genetic  algorithms  to  quickly  explore  the  design 
solution  space.  While  SECTM  cannot  be  validated  until  implemented 
by  acquisition  programs,  it  is  expected  to  increase  the  likelihood  of  suc¬ 
cessful  programs  that,  if  sufficiently  resourced,  can  deliver  on  time  and 
on  budget. 
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Importance  of  Early  Information 

Within  the  increasingly  constrained  fiscal  environment  in  which  the 
DoD  must  operate,  program  life-cycle  cost  control  is  especially  important. 
All  DoD  programs,  no  matter  which  ACAT  level  is  involved,  follow  a  pro¬ 
gram  path  that  has  an  impact  on  life-cycle  costs.  Smaller  ACAT  programs 
can  streamline  or  skip  minor  steps,  but  the  overall  acquisition  process  is 
the  same.  The  Defense  Acquisition  University  has  defined  life-cycle  cost 
across  the  various  program  milestones  as  shown  in  Figure  1  (Defense 
Systems  Management  College,  1990).  Only  10  percent  of  the  program’s 
life-cycle  cost  is  invested  during  the  system’s  research  and  development 
phase  up  to  the  system’s  initial  operational  capability;  however,  this  may 
be  the  most  important  10  percent  of  the  system’s  life-cycle  cost.  As  this 
phase  commits  70  percent  of  the  program’s  life-cycle  costs,  focusing 
significant  time  and  effort  to  assure  that  all  alternatives  are  considered 
is  very  important. 


FIGURE  1.  DEFENSE  ACQUISITION  UNIVERSITY  DEFINED  LIFE- 
CYCLE  COSTS  ACROSS  VARIOUS  PROGRAM  MILESTONES 


Life-Cycle  Cost 


Operations  and  Support 


System  Acquisition 


System 

Research 

and 

Development 


10%  V 


Production 


. .*  „* 

X 

30%  /  \ 

♦ 

♦ 


60% 


♦ 

♦ 

♦ 

♦ 


t  fit 

A  B  C  IDC 
MILESTONES 


YEARS 


287 


Defense  ARJ,  October  2013,  Vol.  20  No.  3:283-308 


Improving  Program  Success  through  Systems  Engineering 


Current  Analysis  of  Alternatives 

In  today’s  environment,  program  managers  are  encouraged  to 
move  as  quickly  as  possible  to  meet  urgent  operational  requirements, 
replacement  schedules,  or  to  save  time.  Because  the  majority  of  the  pre- 
Milestone  B  work  is  level  of  effort,  shortening  this  effort  is  easier  than 
shortening  the  design  and  fabrication  work.  While  this  approach  may 
be  appealing  to  many  program  managers  and  requirements  officers, 
the  acquisition  efforts  leading  to  Milestone  B  set  the  foundation  for  the 
program.  The  work  in  this  phase  defines  the  acquisition  strategy  and 
life-cycle  cost. 

In  2009,  GAO  analyzed  32  major  defense  acquisition  program 
starts  since  fiscal  year  2003.  That  analysis  is  summarized  in  Table  1 
(GAO,  2009a). 


TABLE  1.  COMPARISON  OF  THE  SCOPE  OF  ALTERNATIVES 
CONSIDERED  WITH  PROGRAM  COST  AND  SCHEDULE  GROWTH 


Number  of  programs 
with  cost  or  schedule 
growth3 

Scope  of  alternatives11 

Low 

Moderate 

High 

No  AoA  conducted 

7 

0 

3 

AoA  included  broad  scope 

of  alternatives 

7 

i 

1 

AoA  included  narrow 

scope  of  alternatives 

4 

l 

8 

Source:  GAO. 

a  Cost  growth:  High  =  25  percent  or  greater  growth  in  development  cost  (or  procurement 
costs  for  nondevelopmental  programs)  from  initial  baseline  to  current  estimates, 
Moderate  =  10-24  percent  growth  in  development  cost  (or  procurement  costs  for 
nondevelopmental  programs)  from  initial  baseline  to  current  estimates,  Low  =  less  than 
10  percent  growth  in  development  cost  (or  procurement  costs  for  nondevelopmental 
programs)  from  initial  baseline  to  current  estimates. 

Schedule  growth:  High  =  greater  than  12-month  delay  for  the  initial  operational 
capability  date  or  acquisition  cycle.  Moderate  =  7-  to  12-month  delay  for  the  initial 
operational  capability  date  or  acquisition  cycle,  Low  =  less  than  7-month  delay  for  the 
initial  operational  capability  date  or  acquisition  cycle. 

b  Narrow  scope  of  alternatives  =  2-5  alternative  within  one  concept;  broad  scope  of 
alternatives  =  8-26  alternatives  within  one  concept,  or  alternatives  within  multiple  concepts. 
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Of  the  32  major  DoD  acquisition  programs,  10  programs  did  not 
complete  a  formal  AoA.  For  at  least  seven  of  those,  this  may  have  been 
appropriate  since  they  were  modernization  or  evolutionary  programs. 
The  Defense  Acquisition  Guide,  which  states  that  an  AoA  should  focus  on 
the  end-state  solution,  contains  recommendations  on  a  single  develop¬ 
ment  or  evolutionary  development  path  (Defense  Acquisition  University, 
2012).  The  Milestone  Decision  Authority  can  waive  the  requirement 
for  a  new  AoA  for  incremental  or  modernization  efforts  included  in 
previous  analyses.  The  Navy  Standard  Missile  SM6  is  an  example  of  an 
evolutionary  acquisition  program  where  block  increments  were  used  to 
incrementally  reach  the  final  capability,  thereby  negating  the  necessity 
for  an  AoA.  Thirteen  major  acquisition  programs  conducted  a  narrow 
scope  AoA  where  over  60  percent  of  the  programs  experienced  significant 
cost  or  schedule  growth.  Nine  major  acquisition  programs  conducted  a 
broad  scope  AoA  where  only  one  of  these  programs  experienced  a  sig¬ 
nificant  cost  or  schedule  growth.  This  GAO  (2009a)  study  showed  that 
broader  scope  AoAs  had  less  cost  and  schedule  overruns. 


Because  the  majority  of  the  pre-Milestone  B  work  is 
level  of  effort,  shortening  this  effort  is  easier  than 
shortening  the  design  and  fabrication  work. 


Let  us  highlight  one  program’s  AoA  process.  The  Air  Force  needed 
to  replace  its  KC-135  tanker.  This  was  a  high-visibility  major  ACAT  I 
defense  program  for  the  Air  Force.  The  KC-135  provided  80  percent  of 
U.S.  air  refueling  capability  that  enabled  airpower  to  be  deployed  and 
sustained  overseas  in  a  timely  manner.  The  fleet  of  KC-135s  was  reaching 
50  years  of  age  and  becoming  increasingly  costly  to  maintain  and  oper¬ 
ate.  The  replacement  program  for  these  aircraft  was  expected  to  be  close 
to  $200  billion  (RAND,  2006),  and  6  months  were  allocated  for  the  AoA. 

The  KC-135  AoA  was  required  to  study  the  amount  of  fuel  the  air¬ 
craft  could  supply  along  with  the  times  and  locations  in  a  set  of  mission 
scenarios  (RAND,  2006).  The  AoA  met  these  criteria  through  analyzing 
four  major  aircraft  classes  and  seven  different  methods  to  procure  those 
classes.  However,  the  AoA  was  focused  on  only  one  major  objective:  life- 
cycle  cost.  The  AoA  assumed  that  all  threshold  requirements  must  be 
met,  so  no  analysis  was  conducted  to  see  if  any  single  requirement  was 
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driving  the  cost.  In  addition,  the  AoA  did  not  look  at  the  technology  risk 
of  the  program  to  predict  the  level  of  uncertainty  that  can  drive  program 
overruns  late  in  the  design. 

The  GAO  (2009a)  report  found  that  narrowing  the  AoA  scope  to 
life-cycle  cost  did  not  enable  the  identification  of  the  most  promising 
alternative,  and  reducing  the  AoA  schedule  did  not  allow  enough  time  to 
complete  a  thorough  analysis.  The  GAO  study  recommended  that  DoD 
develop  guidance  for  conducting  robust  AoAs  to  adequately  select  an 
alternative  (GAO,  2009a). 

Systems  Engineering  in  the 
Pre-Milestone  B  Acquisition  Phase 

Program  managers  and  resource  sponsors  are  under  increasing  pres¬ 
sure  to  perform  at  a  higher  level  with  less  resources.  It  appears  unlikely 
that  increasing  either  the  timeline  or  the  cost  of  conducting  an  AoA  is  an 
option.  We  propose  to  use  our  SECTM  in  the  AoA  process  to  thoroughly 
evaluate  additional  alternatives  in  the  same  AoA  timeline. 

The  systems  that  DoD  acquires  have  become  more  complicated,  and 
quantifying  the  effect  that  each  requirement  has  on  these  systems  is 
becoming  increasingly  difficult.  As  the  DoD  strives  to  adopt  more  com¬ 
mercial  practices,  it  will  need  to  adjust  its  acquisition  processes.  Unlike 
the  DoD,  the  commercial  industry  focuses  on  the  market  and  the  price 
point  to  enter  into  that  market.  Most  commercial  industry  program/ proj¬ 
ect  managers  attempt  to  find  the  best  value  for  the  customer  by  providing 
the  most  capability  for  a  set  price.  In  today’s  shrinking  defense  budget,  a 
more  commercial  strategy  may  be  needed  to  keep  the  same  force  levels 
and  capability  despite  reduced  funding.  According  to  Navy  Admiral 
Jonathan  Greenert  (Chief  of  Naval  Operations,  2012): 

We  can  no  longer  afford,  strategically  or  fiscally,  to  let  the  perfect 
be  the  enemy  of  the  good— or  the  good  enough— when  it  comes  to 
critical  war  fighting  capability,  (p.  7) 

Systems  engineering  provides  the  rigor  needed  to  handle  the  increas¬ 
ing  complexity  of  today’s  DoD  systems.  We  are  moving  from  lowest 
cost  for  a  set  threshold  performance  to  simultaneously  minimizing  or 
maximizing  multiple  objectives  like  minimizing  cost,  maximizing  per¬ 
formance,  and  minimizing  program  risk.  In  multiple-objective  analysis. 
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multiple  solutions  exist  that  are  each  optimal,  since  they  are  at  least  as 
good  as  any  other  solution  for  some  weighted  combination  of  the  multiple 
objectives.  For  that  reason,  these  solutions  are  referred  to  as  nondomi- 
nated,  as  they  each  have  no  other  solution  that  dominates  for  at  least 
one  weighted  combination  of  the  objectives.  The  set  of  all  nondominated 
solutions  is  referred  to  as  the  Pareto  front.  Figure  2  is  an  example  of  a 
Pareto  or  nondominated  solution  where  the  design  solutions  are  shown 
in  orange  and  the  optimal  solutions  form  a  line  shown  in  green  (Brown, 
2003).  In  a  Pareto  optimal  designed  system,  the  design  can  trade  off 
cost  versus  risk  to  find  an  optimal  solution.  As  parameters  are  varied  in 
one  optimal  solution,  they  create  other  optimal  solutions  if  the  solution 
improves  in  meeting  at  least  one  objective.  Therefore,  for  a  solution  to  be 
optimum,  it  can  only  decrease  cost  to  the  point  where  it  has  a  negative 
effect  on  performance  or  program  risk. 


FIGURE  2.  SYSTEMS  ENGINEERING  CONCEPT  TOOL  AND 
METHOD  (SECTM) 
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Using  multiple-objective  analysis,  we  developed  a  new  tool  and 
designed  a  method  (SECTM)  to  address  GAO’s  recommendation  that 
DoD  AoAs  need  to  investigate  a  broader  scope  of  alternatives  to  increase 
their  robustness  (GAO,  2009a).  The  SECTM  increases  breadth  of  the 
alternatives  considered,  investigates  program  risk  based  on  technology 
selections,  and  addresses  systems  engineering  complexity  in  the  cost 
estimate,  as  shown  in  Figure  3.  The  proposed  approach  assesses  the 
technologies,  defines  the  system  metrics,  provides  a  tool  to  evaluate  the 
alternatives,  and  can  provide  the  stakeholders  with  an  assessment  of 
optimal  alternatives.  The  alternatives  that  appear  to  be  within  the  avail¬ 
able  resources  could  proceed  to  the  formal  DoD  AoA  process. 

The  steps  in  our  proposed  process  are  detailed  in  the  following 
discussion. 

FIGURE  3.  EXAMPLE  OF  A  PARETO  OR  NON  DOMINATED  SOLUTION 
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Step  One— Assess  Availability  of  Current  Technology 

The  first  step  in  the  proposed  approach  is  to  assess  the  current 
technology  available  and  develop  models  of  those  key  technologies  or 
subsystems.  This  will  allow  a  wide  net  to  be  cast  for  investigating  tech¬ 
nologies,  typically  using  Technology  Readiness  Levels  (TRL)  1-9.  In 
2001,  the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology 
endorsed  the  use  of  TRL  in  new  major  acquisition  programs.  DoD 
Instruction  (DoDI)  5000.02  describes  TRLs  from  a  systems  perspective 
and  states  that  they  are  to  be  used  for  both  hardware  and  software  (DoD, 
2008).  While  to  date,  they  have  been  used  in  the  criteria  for  gate  reviews, 
they  do  not  fold  in  program  risk.  Subsystem  concept  models  should  be 
created  to  represent  system  performance,  cost,  and  risk,  and  include 
TRL  evaluation. 


The  cost  of  doing  systems  engineering  is  becoming 
a  significant  cost  factor  due  to  the  increased 
complexity  of  today’s  systems. 


Step  Two— Define  Objectives  for  Alternatives 

The  second  step,  which  can  occur  in  parallel  to  step  1,  is  the  defini¬ 
tion  of  high-level  objectives  (and  associated  metrics)  for  the  alternatives. 
For  robustness,  there  should  be  at  least  three  primary  objectives  consid¬ 
ered:  technical  performance,  cost,  and  risk  (GAO,  2001).  These  objectives 
will  be  used  to  rank  the  different  alternatives  and  provide  recommenda¬ 
tions  on  the  set  of  optimal  solutions  in  the  next  step. 

Technical  performance  and  cost  objectives  are  part  of  the  standard 
AoA  process  and  should  continue  to  be  defined.  In  addition  to  cost  and 
performance,  we  recommend  using  Technology  Maturity  Risk  as  a  new 
objective.  The  GAO  (2009a)  report  states  that  inadequate  technology 
maturity  is  a  key  factor  in  program  cost  and  schedule  overruns.  The  time 
has  come  to  explicitly  consider  Technology  Maturity  Risk  in  the  AoA  to 
increase  program  success. 

Researchers  at  the  University  of  Southern  California  Center  for 
Software  Engineering  have  proposed  an  approach  that  includes  technol¬ 
ogy  maturity  risk.  They  report  that  TRL  maturity  has  both  positive  and 
negative  aspects.  Higher,  more  mature  technologies  can  have  a  greater 
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risk  of  obsolescence  or  the  possibility  of  a  leap-ahead  technology  dur¬ 
ing  the  life  of  the  DoD  system.  Lower,  less  mature  technologies  have 
greater  development  cost  and  schedule  risk.  These  researchers  have 
proposed  a  new  Technology  Maturity  Risk  function  based  on  the  TRL 
of  a  technology,  the  maturity  of  the  technology  in  a  system,  and  the  risk 
of  obsolescence.  While  Technology  Maturity  Risk  has  been  considered 
in  the  past,  Valerdi  developed  a  new  model  that  links  the  Technology 
Maturity  Risk  to  a  programmatic  cost  (Valerdi,  Boehm,  &  Reifer,  2003). 
From  Valerdi’s  study,  which  included  efforts  of  over  40  systems  engineer¬ 
ing  experts,  this  Technology  Maturity  Risk  has  also  been  associated  with 
a  program  cost  impact.  Table  2  shows  the  rating  scale  for  Technology 
Maturity  Risk  (Valerdi  &  Kohl,  2004). 

Many  programs  underestimate  the  cost  of  the  large  systems  engi¬ 
neering  effort  required  to  develop  complex  systems.  Therefore,  in 
addition  to  the  life-cycle  cost  models  of  the  individual  systems  (aircraft, 
ship,  vehicle,  weapons,  information  technology,  etc.),  the  cost  models 
need  to  consider  the  systems  engineering  cost.  While  advocating  no 
particular  cost  modeling  tool,  the  authors  surmise  that,  to  properly 
determine  life-cycle  costs,  systems  engineering  costs  must  be  con¬ 
sidered  in  the  life-cycle  cost  calculation.  In  2003,  Valerdi  developed 
the  Constructive  Systems  Engineering  Cost  Model  (COSYSMO)  for 
the  purpose  of  estimating  the  systems  engineering  effort  needed  for 
large  complex  systems  (Valerdi,  Boehm,  &  Reifer,  2003).  His  analysis  is 
based  on  four  categories:  product,  platform,  personnel,  and  project  (with 
technology  risk  being  a  driver).  With  assistance  from  the  International 
Council  on  Systems  Engineering,  the  COSYSMO  model  has  been  vali¬ 
dated  with  industrial  partners  while  new  lessons  learned  are  continually 
incorporated  (Valerdi,  Rieff,  Roddler,  &  Wheaton,  2007).  The  cost  of 
doing  systems  engineering  is  becoming  a  significant  cost  factor  due  to 
the  increased  complexity  of  today’s  systems. 

Step  Three— Apply  a  Systems  Engineering  Concept  Tool 

The  heart  of  SECTM  is  our  Systems  Engineering  Concept  Tool. 
The  subsystems  models  from  step  1  and  the  objectives  from  step  2  feed 
our  Systems  Engineering  Concept  Tool.  We  recommend  that  a  genetic 
algorithm  solver  (see  further  discussion)  be  used  because  the  user  can 
select  the  number  of  alternatives  to  be  considered,  and  genetic  algo¬ 
rithms  provide  a  good  estimate  of  the  optimal  solution  set.  For  complex 
systems  with  10  or  more  critical  design  parameters,  the  number  of  dif¬ 
ferent  solutions  can  range  from  10  to  100  billion,  which  is  far  too  many  to 
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investigate.  A  genetic  algorithm  solver  quickly  defines  the  solution  space 
and  provides  a  near-optimal  solution  set  in  a  small  number  of  iterations 
(Deb,  2001).  A  classical  optimization  problem  would  compare  each  pos¬ 
sible  solution  pairwise,  for  which  there  may  be  10  billion  comparisons. 
Genetic  algorithms  can  provide  a  good  estimation  of  the  optimal  solu¬ 
tions  with  a  population  size  as  small  as  100  and  converge  as  quickly  as 
10  generations  from  the  results  of  the  UUV  example  described  later.  Deb 
(2001),  a  recognized  expert  in  genetic  algorithms,  states  that  genetic 
algorithms  have  tremendous  advantage  over  classical  search  techniques 
because  genetic  algorithms  move  the  entire  optimal  population  toward 
the  optimal  solutions  instead  of  a  single  solution. 


Genetic  algorithms  were  developed  to  imitate 
the  processes  that  evolve  in  living  beings,  and 
the  algorithms  allow  the  designs  to  evolve  each 
generation  to  better  meet  the  identified  objectives. 


Genetic  algorithms  were  developed  to  imitate  the  processes  that 
evolve  in  living  beings,  and  the  algorithms  allow  the  designs  to  evolve 
each  generation  to  better  meet  the  identified  objectives.  Even  if  they 
do  not  use  this  formal  methodology,  designs  typically  evolve  just  with 
trial  and  error  (Eddy  &  Lewis,  2001).  The  heart  of  genetic  algorithm 
research  began  with  Schaffer  in  the  1980s.  Now,  many  genetic  algo¬ 
rithms  are  available  that  can  be  applied  to  this  problem  due  to  prior 
research  (Schaffer  &  Grefenstette,1985;  Zitzler,  Deb,  &  Thiele,  2000; 
Horn,  1997).  A  few  common  algorithms  include  the  Vector  Evaluated 
Genetic  Algorithm  (VEGA),  Nondominated  Sorting  Genetic  Algorithm 
(NSGA  II),  and  Niched  Pareto  Genetic  Algorithm  II  (NPGA)  (Zitzler, 
Deb,  &  Thiele,  2000).  VEGA,  one  of  the  first  Pareto  genetic  algorithms 
from  the  1980s,  works  by  assigning  a  randomly  selected  single  objective 
to  each  member  of  the  population.  NPGA  was  developed  by  Horn  and 
Nafpliotis  in  1993,  and  improved  on  the  selection  process  determin¬ 
ing  the  dominance  of  randomly  selected  groups  of  individuals  in  the 
population  (Coello  Coello,  2000).  NSGA  II  was  developed  from  work  by 
Srinivas  and  Deb  (1994)  in  2000,  and  improved  upon  the  basic  genetic 
algorithm  by  sorting  the  population  in  multiple-level  solutions,  starting 
with  the  nondominated  and  binning  them  into  levels  of  domination  until 
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all  solutions  are  binned.  The  level  of  domination  is  used  to  modify  the 
fitness  of  individuals.  This  allows  for  quick  and  computationally  efficient 
algorithms  compared  to  other  methods  (Coello  Coello,  2000),  which  is 
why  we  selected  NSGA II  for  this  research.  Genetic  algorithms  have  been 
used  to  solve  many  complex  problems,  especially  in  aircraft  and  ship¬ 
building  where  many  objectives  compete  with  each  other.  Figure  2  shows 
an  example  of  Pareto  front  trading  off  effectiveness  or  performance  ver¬ 
sus  cost.  The  feasible  region  is  the  large  number  of  possible  alternatives 
in  orange.  The  nondominated  solutions  are  the  set  of  optimal  solutions. 
For  multiple-objective  problems,  two  solutions  are  possible:  one  domi¬ 
nates  (or  is  better  than  the  other)  or  nondominated  (each  solution  is 
equally  good  as  one  another).  Defined  by  Goldberg  (1989),  a  solution  is 
considered  nondominated  when  an  objective  cannot  be  increased  with¬ 
out  reducing  the  other  objectives.  In  complex  systems,  rarely  is  there  one 
optimal  solution,  but  rather  a  set  of  optimal  solutions.  The  stakeholders 
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need  to  make  the  final  choices  between  optimal  solutions  that  best  meet 
their  needs.  The  goal  of  our  Systems  Engineering  Concept  Tool  is  to 
identify  those  optimal  solutions. 

The  Systems  Engineering  Concept  Tool  should,  at  a  minimum, 
include  three  primary  objectives:  technical  performance,  cost,  and  risk. 
Many  AoAs  today  only  consider  the  cost  to  meet  the  threshold  require¬ 
ment,  whereas  SECTM  would  allow  key  performance  parameters  to 
be  separate  objectives,  and  cost  and  risk  could  be  traded  among  those 
key  parameters. 

Step  Four— Presentation  of  Optimal  Alternatives 
to  Stakeholders 

The  fourth  step  in  the  proposed  process  shown  in  Figure  3  is  to  pres¬ 
ent  the  optimal  alternatives  developed  from  the  Systems  Engineering 
Concept  Tool  to  the  stakeholders.  Since  one  optimal  solution  is  rarely 
applicable  in  complex  systems,  an  Executive  Steering  Group  (ESG) 
should  narrow  the  set  of  optimal  solutions  to  those  that  fall  within  the 
resources  available  and  the  program  constraints.  Through  the  use  of 
tradeoffs,  SECTM  will  provide  a  set  of  optimal  solutions  that  meet  the 
metrics  defined  in  step  2.  Since  these  solutions  are  equal  mathemati¬ 
cally,  the  ESG  needs  to  identify  or  narrowthe  “best  solutions”  dependent 
on  preferences  and  experience  (Faulkenberg  &  Wiecek,  2010).  These 
narrowed  solutions  should  then  undergo  a  detailed  analysis  by  subject 
matter  experts.  The  ESG  can  also  decide  to  change  the  metrics  to  refine 
this  analysis  if  none  of  the  alternatives  are  appropriate. 

Step  Five— Detailed  Analysis  (Similar  to  DoD’s 
AoA  Process) 

The  last  step  is  to  take  the  narrowed  set  of  optimal  alternatives  and 
complete  a  detailed  analysis  of  each  alternative.  This  step  is  similar  to 
the  DoD  AoA  process,  which  uses  a  set  of  subject  matter  experts  and 
increased  fidelity  models  and  simulations  to  determine  and  subsequently 
recommend  the  best  alternatives.  The  analysis  in  step  5  will  use  high- 
fidelity  physics  models  that  are  significantly  more  detailed  than  the 
subsystem  concept  models  used  in  step  1. 
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Summary  of  the  Five  Steps 

The  developed  methodology  described  in  the  preceding  5  steps  is 
anticipated  to  increase  the  robustness  of  the  DoD  pre-Milestone  B  Phase 
AoA  by: 

•  Widening  the  solution  space  investigated  within  the  time 
and  personnel  constraints; 

•  Incorporating  the  Technology  Maturity  Risk;  and 

•  Incorporating  the  cost  to  mitigate  Technology  Maturity 
Risk  and  the  level  of  systems  engineering  needed  for  com¬ 
plex  DoD  programs. 

Application  of  SECTM  to 
Unmanned  Systems  Concepts 

In  this  section  we  demonstrate  the  use  of  SECTM  on  unmanned 
systems  that  have  a  strong  appeal  in  the  DoD  environment.  Unmanned 
systems  take  the  DoD’s  most  valuable  asset— its  personnel— and  remove 
them  from  dull,  dirty,  and  dangerous  tasks.  These  systems  demonstra¬ 
bly  reduce  the  forward  deployments  of  our  military  personnel,  thereby 
increasing  the  quality  of  life  for  our  soldiers  and  their  families. 

Unmanned  aerial  vehicles  (UAV)  have  been  used  for  many  years 
with  high  success  in  the  war  against  terrorism.  Secretary  of  the  Navy 
Ray  Mabus  stated  his  priority  in  maintaining  the  competitive  edge  by 
moving  beyond  pilotless  UAVs  to  fielding  unmanned  undersea  vehicles 
(UUV),  as  well  as  surface  vehicles  (Mabus,  2010).  UUVs  will  provide  a 
new  capability  without  significant  experience  or  analysis  to  bound  AoA 
scope.  Since  UUVs  may  be  the  next  big  acquisition  of  unmanned  systems, 
they  are  a  good  test  case  for  SECTM.  The  following  discussion  reapplies 
the  steps  defined  earlier  using  an  analysis  of  UUV  designs  as  an  example. 

Step  One— Assess  Availability  of  Current  Technology 

The  first  step  in  applying  SECTM  was  to  analyze  the  UUV  sub¬ 
systems  and  to  determine  the  critical  technologies  in  each  subsystem. 
Subsystem  models  were  created  from  core  hydrodynamic  texts  and  UUV 
literature  from  Massachusetts  Institute  of  Technology  and  Southampton 
Universities  (Furlong,  McPhail,  &  Stevenson,  2007).  We  developed  a 
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basic  UUV  cost  model  using  the  Naval  Sea  Systems  Command  cost 
estimation  handbook,  and  adapting  a  submarine  cost  model  and  systems 
engineering  cost  models  (Valerdi,  Boehm,  &  Reifer,  2003). 

We  identified  critical  technologies  for  achieving  endurance  that 
were  based  on  experience  with  the  Autosub  UUV  (Furlong,  McPhail,  & 
Stevenson,  2007)  being  designed  for  an  endurance  of  5,000  meters  and 
buoyancy-driven  UUVs.  For  simplicity  of  this  example,  the  two  driving 
technologies  are  the  energy  density  of  the  primary  power  system  and  the 
hotel  load  (for  example  heating,  computing,  power  distribution).  Most 
current  UUV  systems  use  batteries,  which  are  a  high  TRL  (mature), 
but  low-energy  density.  However,  high-energy  density  power  systems 
like  fuel  cells  and  combustors  are  being  developed  and  show  promise 
at  low  TRLs.  We  completed  a  market  survey  to  look  at  the  different  bat¬ 
tery  technologies  and  their  energy  density  as  a  function  of  TRL.  Hotel 
power  was  linked  to  one  primary  technology— computer  processors.  We 
used  current  quad-core  processors  as  the  standard  processor  at  TRL  8. 
New  gaming  and  cell  core  processors  are  being  developed  that  have  the 
potential  to  reduce  the  processing  power  by  a  factor  of  four,  but  these  are 
only  at  TRL  3.  Once  again,  we  completed  a  market  survey  and  created  a 
model  to  link  processing  power  with  a  TRL. 

Step  Two— Define  Objectives  for  Alternatives 

Step  Two  defines  the  objectives  for  the  system  Pareto  analysis. 
We  used  the  2004  Navy  UUV  Master  Plan  as  a  guiding  document  to 
determine  the  UUV  design  objectives  (Department  of  the  Navy,  2004). 
The  first  objective  was  to  maximize  the  endurance  or  range  of  a  UUV  to 
be  able  to  perform  Navy  missions  like  mine  warfare  and  intelligence, 
surveillance,  and  reconnaissance.  The  second  design  objective  was  to 
minimize  the  UUV’s  volume.  This  is  important  for  integration  of  the 
UUV  onto  existing  Navy  platforms  since  larger  UUVs  may  not  fit  on 
many  Navy  ships.  The  third  and  fourth  objectives  were  cost  and  tech¬ 
nology  risk. 
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Figure  4  illustrates  the  10  different  parameters  we  considered  in  this 
UUV  analysis  and  links  those  parameters  to  each  objective. 

FIGURE  4.  SECTM  OBJECTIVES  AND  SYSTEM  PARAMETERS 

Four  design  objectives  to  optimize: 

Range  =  f  (  I,  d,  B,  V,  PH,  nm,  np)  Maximize 
Volume  =  f  (  I,  d,  B,  PH,  D,  VM,  o)  Minimize 
Cost = f  ( I,  d,  D,  B,  TRLs)  Minimize 
Technology  Risk = f  ( TRLs)  Minimize 


Design  Parameters  for  UUVs: 

1.  Length  (1) 

5.  Velocity  (V) 

9.  Propulsion  Efficiency  (np) 

2.  Diameter  (d) 

6.  Maximum  Velocity  (VM) 

10.  Depth  (D) 

3.  Energy  Density  (B) 

7.  Hotel  Power  (PH) 

4.  Foam  Specific  Gravity  (a) 

8.  Motor  Efficiency  (r)J 

Early  communication  with  stakeholders  on 
potential  alternatives  can  facilitate  a  better 
understanding  of  the  requirements. 


Step  Three— Apply  a  Systems  Engineering  Concept  Tool 

We  chose  the  NSGA-II  genetic  algorithm  developed  by  Deb  (Deb, 
Pratap,  Agarwal,  &  Meryarivan,  2002)  for  our  basic  genetic  algorithm 
solver  because  of  this  algorithm’s  computational  efficiency.  NSGA-II’s 
computational  efficiency  can  be  approximated  by  the  formula:  /(MV) 
as  opposed  to  other  sorting  algorithms,  which  us ef(M*N3)  where  M  is 
the  number  of  objectives  and  N  is  the  genetic  population  size.  For  the 
UUV  example  where  M=  3  and  IV  =  100,  NSGA-II  saved  2,970,000  com¬ 
putations.  We  programmed  equations  for  each  of  the  design  objectives 
into  Matrix  Laboratory,  or  MATLAB  programming  language  using  the 
NSGA-II  algorithm  for  the  optimization. 
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Since  the  population  size  is  variable,  it  can  be  selected  by  the  users. 
Increased  population  size  provides  more  points  on  the  Pareto  front  to 
better  identify  design  trends.  However,  increased  population  size  will 
square  the  number  of  computations  required. 

For  this  example,  there  are  10  basic  design  parameters  (r).  For  simpli¬ 
fication,  if  each  design  parameter  had  only  10  different  applicable  values 
( n ),  then  through  pairwise  comparison: 

Estimated  number  of  comparison  =  nr 

For  this  example,  n=  10  and  r  =  10,  which  is  10  billion  combinations 
that  would  have  to  be  analyzed.  This  is  far  too  many  to  accomplish  in  just 
a  few  months;  however,  the  use  of  a  genetic  algorithm  solver  reduces  the 
number  of  processes  significantly.  The  genetic  solver  starts  with  random 
solutions.  Those  solutions  that  have  a  higher  match  to  the  objectives  are 
selected  for  regeneration  and  combined  together  to  create  a  new  genera¬ 
tion.  This  process  is  continued  and  mimics  the  way  living  species  survive 
and  adapt  to  the  environment.  Genetic  algorithms  usually  can  converge 
in  10  generations;  therefore,  the  amount  of  calculation  needed  is  the 
population  size  times  the  number  of  generations: 

Number  of  designs:  population  x  generations  =  100  x  10  =  1,000 

The  results  of  this  application  show  that  for  the  UUV  design  dis¬ 
cussed  here,  which  is  fairly  simple  compared  to  many  DoD  systems, 
the  solution  space  of  10  billion  different  design  combinations  can  be 
approximated  by  a  population  size  of  10  or  100  with  less  than  1,000 
design  iterations  using  the  SECTM.  SECTM  is  a  very  efficient  way  to 
determine  a  set  of  optimal  alternatives  to  present  to  the  stakeholders. 

Early  communication  with  stakeholders  on  potential  alternatives 
can  facilitate  a  better  understanding  of  the  requirements.  Today’s 
systems  are  so  complex  and  highly  integrated,  that  it  is  impossible  to 
understand  the  large  impacts  that  small  changes  can  make  without  the 
use  of  analysis  tools.  SECTM  provides  a  visualization  of  the  tradeoffs  of 
risk,  cost,  and  technical  performance.  These  tradeoffs  are  very  important 
to  the  success  of  the  program.  Using  current  practices,  stakeholders  are 
not  presented  with  enough  data  to  make  good  decisions.  Steps  4  and  5 
were  not  completed  in  this  example  as  they  feed  the  DoD  AoA  process 
and  were  not  needed  to  show  the  feasibility  of  the  SECTM. 
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Conclusions 

In  today’s  reduced  budget  and  constrained  fiscal  environment,  mak¬ 
ing  acquisition  decisions  that  provide  the  best  value  to  the  nation’s  armed 
forces  and  the  DoD  is  extremely  important.  Over  70  percent  of  a  system’s 
life-cycle  cost  is  determined  by  Milestone  B;  therefore,  the  largest  impact 
can  be  made  during  these  early  program  stages.  Unfortunately,  this  is 
where  a  large  majority  of  programs  streamline,  reduce,  or  cut  activities  to 
save  time  and  funding.  Out  of  32  programs  reviewed  by  the  GAO  (2009a), 
60  percent  of  the  programs  that  completed  limited  scope  in  their  AoAs 
experienced  significant  cost  and/or  schedule  overruns  compared  to  less 
than  10  percent  in  those  programs  that  completed  a  robust  AoA. 

Applying  the  SECTM  in  the  pre-Milestone  B  Acquisition  Phase  is  an 
option  to  increase  the  AoA’s  robustness  without  significantly  increasing 
cost  or  time.  Current  processes  use  a  team  of  experts  to  analyze  a  few 
predetermined  alternatives  (three  to  10  for  a  typical  acquisition  pro¬ 
gram)  and  primarily  conduct  interviews  to  make  subjective  analyses. 
This  article  proposed  a  SECTM  to  be  used  in  the  AoA  process  to  help 
determine  or  down-select  the  few  alternatives  that  are  investigated 
in  depth  by  an  AoA  team.  When  we  applied  our  SECTM  to  a  UUV,  we 
were  able  to  reduce  10  billion  design  combinations  to  a  set  of  only  a  few 
optimal  solutions.  This  initial  systems  engineering  step  can  be  done 
rapidly  using  modeling  and  simulation  tools,  and  by  using  the  engineer¬ 
ing  process  to  down-select  the  alternatives  instead  of  a  steering  group 
committee  process. 

This  article  also  presented  the  importance  of  the  pre-Milestone  B 
Acquisition  Phase  in  setting  the  foundation  for  the  success  of  the  pro¬ 
gram.  This  methodology  presented  a  way  to  increase  the  robustness  of 
the  alternatives  considered  in  pre-Milestone  B  acquisition  documenta¬ 
tion  (primarily  AoAs)  and  incorporates  the  following  new  aspects: 

•  Widening  the  solution  space  investigated  within  the  time 
and  personnel  constraints; 

•  Incorporating  the  Technology  Maturity  Risk;  and 

•  Incorporating  the  cost  to  mitigate  Technology  Maturity 
Risk  and  the  level  of  systems  engineering  for  complex  DoD 
programs. 
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While  pre-Milestone  B  efforts  only  account  for  less  than  10  percent 
of  the  total  life-cycle  cost,  they  are  the  most  important  10  percent  of 
funding  because  they  set  the  acquisition  program  on  a  sound  foundation 
and  business  case.  Errors  in  this  phase  cost  between  three  and  10  times 
more  to  fix  in  later  phases.  The  GAO  recommended  to  the  DoD  that  new 
criteria  should  be  set  for  execution  of  AoAs,  with  the  DoD  agreeing  to 
the  recommendation.  The  approach  proposed  in  this  article  is  a  way  to 
increase  the  robustness  of  DoD’s  AoAs. 


Author  Biographies 


Dr.  Daniel  Deitz  is  an  unmanned  undersea 


systems  subject  matter  expert  at  the  Office 
of  Naval  Research  with  13  years  of  Department 
of  Navy  acquisition  professional  experience. 
He  holds  an  MS  in  Systems  Engineering  from 
Johns  Hopkins  University,  a  BS  in  Mechanical 
Engineering  from  Pennsylvania  State 
University,  and  a  PhD  from  The  George 
Washington  University.  Dr.  Deitz  holds  several 
Defense  Acquisition  Workforce  Improvement 
Act  certifications. 

(E-mail  address:  daniel.deitz@navy.mil) 

Dr.  Timothy  J.  Eveleigh  is  an  adjunct  pro¬ 
fessor  of  Engineering  Management  and 
Systems  Engineering  at  The  George 
Washington  University  and  an  International 
Council  on  Systems  Engineering-certified 
Systems  Engineering  Professional.  He  has 
over  30  years’  industry  experience  working 
DoD/intelligence  community  information 
technology  acquisition  challenges,  research 
and  development,  and  enterprise  architecting. 
Dr.  Eveleigh  has  a  30-year  parallel  career  as 
an  Air  Force  Reserve  intelligence  officer  and 
developmental  engineer,  focused  on  command 
and  control  integration. 

(E-mail  address:  eveleigh@gwu.edu) 


Defense  ARJ,  October  2013,  Vol.  20  No.  3:283-308 


304 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


iDr.  Thomas  H.  Holzer  is  an  adjunct  profes¬ 
sor  of  Engineer  ing  Management  and  Systems 
Engineering  at  The  George  Washington 
University.  He  was  the  director,  Engineering 
Management  Office,  National  Geospatial- 
Intelligence  Agency,  with  over  35  years’ 
experience  in  systems  engineering  and  lead¬ 
ing  large-scale  information  technology 
programs.  Dr.  Holzer  holds  a  Doctor  of 
Engineering  Management  and  Master  of 
Science  in  Engineering  Management  from 
The  George  Washington  University;  and  a 
Bachelor  of  Science  in  Mechanical 
Engineering  from  University  of  Cincinnati. 

(E-mail  address:  holzer@gwu.edu) 

Dr.  Shahryar  Sarkani  is  an  adjunct  profes- 
sor  in  the  Department  of  Engineering 
Management  and  Systems  Engineering  at 
The  George  Washington  University,  with  over 
20  years  of  experience  in  software  engineer¬ 
ing  focusing  on  architecture  and  design.  Dr. 
Sarkani  holds  a  Doctor  of  Science  in  Systems 
Engineering  from  The  George  Washington 
University,  a  Master  of  Science  in  Mathematics 
from  University  of  New  Orleans,  and  a 
Bachelor  of  Science  in  Electrical  Engineering 
from  Louisiana  State  University. 

(E-mail  address:  emseor2003@yahoo.com) 


305 


Defense  ARJ,  October  2013,  Vol.  20  No.  3: 283-308 


Improving  Program  Success  through  Systems  Engineering 


References 

Brown,  A.,  &  Salcedo,  J.  (2003).  Multiple-objective  optimization  in  naval  ship 
design.  Naval  Engineers  Journal,  775(4),  49-61. 

Chief  of  Naval  Operations.  (2012,  March).  FY 2013  Department  of  Navy 
posture.  Statement  of  Admiral  Jonathan  Greenert,  Chief  of  Naval 
Operations,  Before  the  112th  Congress.  Retrieved  from  http://www.navy. 
mil/cno/120316_ps.pdf 

Coello  Coello,  C.  (2000).  An  updated  survey  of  GA-Based  Multiobjective 
Optimization  Techniques.  ACM  Computing  Surveys,  32(2),  109-143. 

Deb,  K.  (2001).  Multi-objective  optimization  using  evolutionary  algorithms. 
Chinster,  England:  Wiley. 

Deb,  K.,  Pratap,  A.,  Agarwal,  S.,  &  Meryarivan,  T.  (2002).  A  fast  and  elitist 
multiobjective  genetic  algorithm:  NSGA  II.  IEEE  Transactions  on 
Evolutionary  Computing,  6(2),  182-197. 

Defense  Acquisition  University.  (2012).  Defense  Acquisition  Guidebook. 

Retrieved  from  https://at.dod.mil/docs/DefenseAcquisitionGuidebook. 
pdf 

Defense  Systems  Management  College.  (1990).  DSMC  systems  engineering 
management  guide.  Retrieved  from  http://oai.dtic. mil/oai/oai?verb=getR 
ecord&metadataPrefix=html&identifier=ADA223168 

Department  of  Defense.  (2008).  Operation  of  the  defense  acquisition  system 
(DoDI  5000.02).  Retrieved  from  http://www.dtic.mil/whs/directives 

Department  of  Defense  Inspector  General.  (2012).  Acquisition  of  the  Navy 
organic  airborne  and  surface  influence  sweep  need  improvement  (Report 
No.  DoDIG-2012-101).  Retrieved  from  http://www.dodig.mil/audit/ 
reports/fyl2/DoDIG-2012-101.pdf 

Department  of  the  Navy.  (2004).  The  Navy  unmanned  undersea  vehicle 
(UUV)  master  plan.  Retrieved  from  http://www.navy.mil/navydata/ 
technology/uuvmp.pdf 

Eddy,  J.,  &  Lewis,  K.  (2001).  Effective  generation  of  Pareto  sets  using  genetic 
programming.  Proceedings  of  DETC'01,  ASME 2001  Design  Engineering 
Technical  Conference  and  Computers  and  Information  in  Engineering 
Conference,  Pittsburgh,  PA,  September  9-12,  2001. 

Faulkenberg,  S.,  &  Wiecek,  M.  (2010).  Quality  discrete  representation  in 
multiple  objective  programming.  Optimization  and  Engineering,  IKS), 
423-440. 

Furlong,  M.,  McPhail,  S.,  &  Stevenson,  P.  (2007).  Concept  for  an  ultra-long- 
range  survey  class  AUV.  Proceedings  of  IEEE  Oceans  2007-Europe 
Conference,  Hampton,  UK,  June  18,  2007. 

Goldberg,  D.  E.  (1989).  Genetic  algorithms  in  search,  optimization,  and 
machine  learning.  Reading,  MA:  Addison-Wesley  Professional. 

Government  Accountability  Office.  (2001).  Joint  Strike  Fighter  acquisition: 
Mature  critical  technologies  needed  to  reduce  risks  (Report  No.  GAO-02- 
039).  Retrieved  from  http://www.gao.gov/new.items/d0239.pdf 


Defense  ARJ,  October  8013,  Vol.  80  No.  3:883-308 


306 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Government  Accountability  Office.  (2009b).  Best  practices:  High  levels  of 
knowledge  at  key  points  differentiate  commercial  shipbuilding  from  Navy 
shipbuilding  (Report  No.  GAO-09-322).  Retrieved  from  http://www.gao. 
gov/new.  items/d09322.  pdf 

Government  Accountability  Office.  (2009a).  Many  analyses  of  alternatives 
have  not  provided  a  robust  assessment  of  weapon  system  options 
(Report  No.  GAO-09-665).  Retrieved  from  http://www.gao.gov/new. 
items/d09665.pdf 

Horn,  J.  (1997).  The  nature  of  niching:  Genetic  algorithms  and  the  evolution 
of  optimal,  cooperative  populations  (Doctoral  dissertation,  University  of 
Illinois  at  Urbana-Champaign).  Retrieved  from  http://citeseerx.ist.psu. 
edu/viewdoc/download?doi=10.1.1.33.8352&rep=repl&type=pdf 

Mabus,  R.  (2010,  May).  The  SECNAV:  Implementing  policy,  taking  care  of 
people.  Seapower  magazine  interview  by  Managing  Editor  Richard  R. 
Burgess  and  Editor  in  Chief  Amy  L.  Wittman.  Retrieved  from  http://www. 
navy.mil/navydata/people/secnav/Mabus/lnterview/Seapowerlnterview- 
Raypercent20Mabus.pdf 

Pincus,  W.  (2012,  June  18).  Smaller  defense  programs  need  a  closer  look,  too. 
The  Washington  Post.  Retrieved  from  http://www.washingtonpost.com/ 
world/national-security/smaller-defense-programs-need-a-closer-look- 
too/2012/06/18/g  JQABqGXmV_story.html 

RAND.  (2006).  Analysis  of  alternatives  (AoA)  for  KC-135  recapitalization. 
Retrieved  from  http://www.rand.org/pubs/monographs/MG495.html 

Schaffer,  J.,  &  Grefenstette,  J.  (1985).  Multi-Objective  learning  via  genetic 
algorithms.  In  A.  K.  Joshi  (Ed.),  Proceedings  of  the  Ninth  International 
Joint  Conference  on  Artificial  Intelligence  (Vol.  1,  pp.  593-595),  Los 
Angeles,  CA,  August  18-23, 1985. 

Srinivas,  N.,  &  Deb,  K.  (1994)  Multiobjective  optimization  using  nondominated 
sorting  in  genetic  algorithms.  Journal  of  Evolutionary  Computing,  2(3), 
221-248. 

Valerdi,  R.,  Boehm,  B.,  &  Reifer,  D.  (2003).  COSYSMO:  A  constructive  systems 
engineering  cost  model  coming  of  age.  Paper  presented  at  the  INCOSE 
13th  Annual  International  Symposium  on  Engineering  Tomorrow's  World 
Today,  Crystal  City,  VA,  June  29-July  3,  2003. 

Valerdi,  R.,  &  Kohl,  R.  (2004).  An  approach  to  technology  risk  management. 
Paper  presented  at  the  Engineering  Systems  Division  Symposium, 
Massachusetts  Institute  of  Technology,  Cambridge,  MA,  March  29-31, 
2004. 

Valerdi,  R.,  Rieff,  J.,  Roddler,  G.,  &  Wheaton,  M.  (2007).  Lessons  learned  from 
industrial  validation  of  COSYSMO.  Paper  presented  at  the  17th  Annual 
INCOSE  International  Symposium  on  Systems  Engineering:  Key  to 
Intelligent  Enterprises,  San  Diego,  CA,  June  24-28,  2007. 

Zitzler,  E.,  Deb,  K.,  &  Thiele,  L.  (2000).  Comparison  of  multi-objective 

evolutionary  algorithms:  Empirical  results.  Evolutionary  Computing,  8(2), 
173-195. 


307 


Defense  ARJ,  October  2013,  Vol.  20  No.  3: 283-308 


